Real-Time Voice-Based Web Form Interaction using Natural Language Processing

Abstract

This project implements an intelligent voice-driven form filling assistant using Streamlit, speech recognition, and natural language processing. The system enables users to interact with online forms through voice commands, dynamically extracting form fields from any URL and guiding users through a conversational interface to complete them. The Voice Based Form Filler system features multi-language translation support (English, Hindi & Telugu), speech synthesis for instructions, efficient form field extraction using Selenium, NLP-based context verification, and automatic form submission capability. It employs caching strategies, input verification algorithms, and field context recognition to maintain responsive performance while handling complex interactions. This solution makes web forms more accessible, especially for users with limited typing ability or who prefer voice interaction over traditional input methods.

Introduction

Digital form completion is a major obstacle for users with disabilities, limited motor skills, or low digital literacy. Traditional keyboard-and-mouse interfaces are not inclusive, while existing voice-based systems often lack accuracy, context awareness, and accessibility features.

Objective

The project aims to develop a real-time, voice-driven form-filling system that improves accessibility and usability using NLP-based context verification, multilingual support, and secure, local processing.

Key Features & Methodology

• Web Automation: Uses Selenium to extract and interact with web forms.

• Speech Recognition: Converts spoken input into text with Google Speech API.

• NLP Context Verification: Uses spaCy to interpret input based on field type (e.g., converting “at” to “@” in emails).

• Multilingual Support: Supports English, Telugu, and Hindi with voice translation.

• Voice Interaction: Reads out field labels and instructions, aiding visually impaired users.

• Error Handling: Tracks Word Error Rate (WER) to identify and correct mistakes.

• Data Export: Saves completed form data in Excel for record-keeping.

Literature Review Highlights

• Usharani et al. (2020): Identified hardware limitations in simple voice-text systems.

• Mani et al. (2021): Showed benefits and accuracy limits of older speech models (HMM).

• Ramasubramanian et al. (2022): Highlighted barriers in systems like UDID registration.

• Gaud et al. (2022): Explored NLP and conversational AI potential for accessibility.

• Syed (2025): Proposed a multi-modal system combining voice and image recognition but found voice-only systems simpler and more reliable in certain cases.

Results

• Achieved 0% WER in tests with 10 repeated input attempts.

• Provided a fully voice-driven interface, enhancing accessibility for visually impaired users.

• Enabled accurate form filling with intelligent field mapping and semantic verification.

Limitations & Future Scope

• Difficulty handling dynamic content and JavaScript-heavy forms.

• Lacks persistent user data for pre-filling common fields.

• Multilingual support is limited to three languages.

• Needs enhanced semantic models for better field classification and regional dialect handling.

Conclusion

This research demonstrates the feasibility and value of voice-driven form completion as an accessibility solution for web interactions. Our implementation shows significant benefits for users with various accessibility needs, reducing completion time and error rates while improving overall satisfaction. The solution presented in this paper can help bridge the digital divide and ensure equitable access to web-based resources.

References

[1] S. Usharani, P. M. Bala and R. Balamurugan, \"Voice Based Form Filling System For Visually Challenged People\", 2020 International Conference on System, Computation, Automation and Networking (ICSCAN), Pondicherry, India, 2020. [2] Mani, V.A., Dhanalakshmi, A., Dharani, S., Bharathi, B. (2021). Speech Enabled Automatic Form Filling System. In: Reddy, V.S., Prasad, V.K., Wang, J., Reddy, K.T.V. (eds) Soft Computing and Signal Processing. Advances in Intelligent Systems and Computing, vol 1325. Springer, Singapore. [3] S. S. Ramasubramanian, S. Koodli, P. S. Nair, M. Sadique and H. Mamatha, \"Voice Assisted Form Filling for the Differently Abled,\" 2022 International Conference on Distributed Computing, VLSI, Electrical Circuits and Robotics (DISCOVER), Shivamogga, India, 2022. [4] Asha Gaud, Bhumi Mota, Dhananjay Kumbhar, Veer Kumar, Prof. Shashank Gothankar, “Chatbot Personal Assistant Using Natural Language Processing (NLP)”, 2022, International Journal Of Innovative Research In Technology, MCT Rajiv Gandhi Institute of Technology, Mumbai, India. April 2022. [5] Waseem Syed \"AI-Powered Multi-Modal Form Filling: Advancing Accessibility through Voice and Image Recognition\", 2025 International Journal of Scientific Research in Computer Science, Engineering and Information Technology, JNTU, Hyderabad, India.

Copyright

Copyright © 2025 Carolina Pasham Frazer. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.